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ABSTRACT: 

Apparatus for routing packets in a communication network comprises a plurality (1 00) of 
per-connection queues (20a-20i), each queue established for receiving packets from a 
respective source and temporarily storing received packets before routing to a particular 
destination; a weighted fair-queuing scheduler (40) for servicing packets from each of the plurality 
of per-connection queues at guaranteed pre-allocated rates; a sensing device for sensing a 
presence or absence of packets in queues, the absence of packets in queues indicating 
availability of excess bandwidth; and, a state dependent scheduler (200) for redistributing 
excess bandwidth upon sensing of queues absent packets, the state dependent scheduler (200) 
servicing those queues in accordance with a state variable corresponding to a performance 
property of the queues, wherein delay and isolation properties for routing packets of respective 
queues in weighted fair-queuing is preserved. 



(19) 



J 



(12) 



(43) Date of publication: 

19.08.1998 Bulletin 1998/34 



Europdisches Patentamt 
European Patent Office 
OKice europ6en des brevets (11) EP 0 859 492 A2 

EUROPEAN PATENT APPLICATION 

(51) int a.^: H04L 12/56. H04L 29/06 



(21) Application number: 98300754.3 

(22) Date of filing: 03.02.1998 



(84) Designated Contracting States: 


(72) 


Inventors: 


AT BE CH DE DK ES R FR GB GR IE IT LI LU MC 




Duff ield, Nicholas G. 


NL PT SE 




Hoboken, New Jersey 07030 (US) 


Designated Extension States: 




Stiliadis, Dimitrlos 


AL LT LV MK RO SI 




Middletown, Monmouth, NJ 07748 (US) 




• 


Lukshman, Tirunellai Viswanathan 


(30) Priority: 07.02.1997 US 37844 P 




Eatontown, IVlonmoutti, New Jersey 07724 (US) 


18.1 1.1997 US 








(74) 


Representative: 


(71) Applicant: 




Budcley, Christopher Simon ThirsIc et al 


LUCENT TECHNOLOGIES INC. 




Lucent Technologies (UIQ Ltd, 


Mun^ Hill, New Jersey 07974-0636 (US) 




5 Mornington Road 

Woodford Green, Essex iG8 OTU (GB) 



CM 
< 
CsJ 

in 

00 

o 

Q. 
LU 



(54) Fair queuing system with adaptive bandwidth redistribution 



(57) Apparatus for routing packets in a communica- 
tion network comprises a plurality (100) of per-connec- 
tion queues (20a~20i). each queue established for 
receiving packets from a respective source and tempo- 
rarily storing received packets before routing to a partic- 
ular destination; a weighted fair-queuing scheduler (40) 
for servicing packets from each of the plurality of per- 
connection queues at guaranteed pre-allocated rates; a 
sensing device for sensing a presence or at>sence of 
packets in queues, the absence of packets in queues 



indicating availability of excess bandwidth; and. a state 
dependent scheduler (200) for redistritxjting excess 
t>andwidth upon sensing of queues at>sent packets, the 
state dependent scheduler (200) sen^k;ing those 
queues in accordance with a state variat>le correspond- 
ing to a performance property of the queues, wherein 
delay arxl isolation properties for routing packets of 
respective queues in weighted fair-queuing is pre- 
served. 
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Description 

Held of the Invention 

The instant invention relates generally to packet- 
based communication systems and particularly to fair- 
queuing systems implemented in routers and switches 
In a packet-based communication system. 

Bgc kgr ^und Of the Invention 

Much research has been devoted to development 
of queuing systems for packet-based comrrunication 
networks that emulates as closely as possOsle, an ideal 
fluid flow" model, i.e., where data packets communi- 
cated from multiple sources are considered to be infi- 
nitely divisible and multiple sources may transmit their 
data simultaneously, e.g., on a single physical commu- 
nication link. Infinite divisibility is not feasible in practice. 
In packet networks, typically once a packet is transmit- 
ted over the linK the whole packet must be sent, i.e., rt 
cannot be interrupted to transfer another packet in 
between. As there exists a desire to provide Quality of 
Service X^S" guarantees in a packet network, there is 
required the implementation of traffic scheduling meth- 
ods in the data packet switches or routers. The functbn 
of a scheduling method is to select, for each outgoing 
link of a switch, the packet to be transmitted In the next 
cycle from the available packets belonging to the com- 
nujnication sessions sharing the output link. This selec- 
tion must be performed such that the QoS guaranteed 
for the individual traffic sessions, e.g.. upper tx>unds on 
maximum delay, are satisfied. Implementation of the 
method may be hardware or software, txjt because of 
speed conskierations, scheduling is usually imple- 
mented in hardware in ATM switches and high-speed 
routers. 

Many dtffererrt scheduling methods have been pro- 
posed to approximate the theoretical scheduling disci- 
pline known as Generalized Processor Sharing (GPS) 
system, which is a discipline defined with respect to the 
fluid" model. Such a GPS would allow for tight control of 
the t>andwidth allocated to each session comnunicating 
on a link. However, as packets transmitted by a session 
cannot be divkied further, tiie data from multiple 
sources must be interleaved only at packet boundaries. 
Thus, the GPS discipline cannot be implemented in 
practice in a packet-switched network. 

Servicing of separate queues by simple FIFO. 
Round Robin, and fair queuing technk^ues, and the like, 
are well-known. However. "Weighted" fair-queuing 
CWFQO schemes have been developed that closely 
approximate tiie fluid system. Particularly. A. Demers. 
S. Keshav, S. Shenker. in tiie reference "Analysis and 
Simulation of a Fair Queuing Algorithm" Internetwork- 
ing: Research and Experience, pp. 3-26. vol. 1, 1990 
describe a fair queuing scheme that emulates GPS by 
essentially simulating a fluid flow GPS system for refer- 



ence and k>asing packet scheduling decisions on the 
order of departures in the GPS system. In weighted fair 
queuing, each traffic session / sharing the output link 
controlled by the scheduling method is assigned a value 
5 (h conesponding to the reserved bandwktth of the ses- 
sion. The values 4>j are computed such that the reserved 
barxlwkfth of session / on the link is given by: 

- — . 

M 

75 where the denominator conputes the sum of the 4>j> val- 
ues for all u sessions sharing the link. 

Particularly, as shown in Fig. 1 , a WFQ system 100 
is provkJed witii a plurality of per-connection queues 
20a 20i. with each queue storing packets in a drffer- 

20 ent portion of a shared merrwry 25 for temporarily stor- 
ing packets of information, e.g., input traffic from a 
source device such as a data terminal. It is understood 
tiiat there can be provkled different types of queues for 
accommodating different types of traffic. e.g.. audio, 

25 video, data. etc. Additionally provided is a shaper 
30a,....30i that forward packets from the queues to the 
Weighted Fair Queueing Server with a rate exactly 
equal to the allocated. The Weighted Fair Queueing 
scheduler assumes that a weight is associated with 

30 each queue 20a,.., 201; respectively, so that the service 
offered by the scheduler to each one of these queues 
while they have packets waiting is always In proportion 
to the weights. For example, let us assume that the 
capacity (bandwidth) of the link C=10 packets/sec. Let 

35 US also assume that the scheduler is sending three 
queues; Ql being accorded a weight WQ1:=20%. 
queque Q2 being accorded a weight WQ2=30%, and 
queue Q3 being accorded a weight WQ3=50%. Then, if 
all queues have packets waiting, then Ql and 02 will 

40 receive a guaranteed bandwidth of 2 and 3 packets/sec- 
ond respectively, and Q3 will receive a guaranteed 
bandwkith of 5 packets/second. However, if, for exam- 
ple. Q3 does not have any packets waiting, then the 
excess bandwkftti is equal to 5 packets/second. In a 

45 WFQ system, this excess bandwidth is redistritHJted in 
proportion to ttie associated weights of the queues that 
have packets waiting. In the atx>ve example, when 
queue Q3 does not have packets waiting, ttie excess 
bandwkith will be distributed proportionally to queues 

50 Q1 and Q2 so. that they now receive instaneous band- 
wkitii of 4 and 6 packets per second respectively. Each 
packet leaving its respective shaper 30 is forwarded 
directly to a Rate Proportional Server 40 ("RPS"). which 
may be any weigtited fair queuing variation, that for- 

55 wards tiie packets to output link 50. 

In such a WFQ scheme. t>ene^icial properties exist 
such as erxl-to-end delay guarantees, e.g., each packet 
is guaranteed a certain rate for each packet fk)w in the 
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Stream, and, the provision of isolation between streanis. 
e.g.. a misbehaving source will not effect the flow of 
other streams. Additionally, an added benefit is that 
when there is underutilization of capacity, e.g., when 
flow Is particularly bursty and there may be Idle time, the 
WFQ system facilitates the redistribution of the unused 
bandwidth so as to preserve work-conservation prop- 
erty. Presently, the redistribution property of unused 
bandwidth capacity anfx}ng the queues is done in a 
manner inherited from the fluid-flow model, e.g.. in 
accordance with the weight associated with the particu- 
lar queue. Thus, when the packet queues are idle, 
"excess" bandwidth is redistributed to bacWogged con- 
nections in proportion to their weights which are based 
on long-term requirements. 

TTie drawback of GPS that all fair queuing systems 
inherit in their dose emulation of GPS is that GPS 
severely restricts state-dependent bandwidth sharing. 
The only state-dependency in GPS is in the number of 
bacMogged connections. There is no further latitude 
and sharing is determined by the guaranteed rates 
which are set based on long term needs of the connec- 
tions. This restriction on bandwidth sharing is more 
stringent tfian that necessary to preserve a key of prop- 
erty of fair queuing, the ability to guarantee worst case 
delay bounds for leaky bucket controlled traffic sources. 
Consequently, there is no need for fair queuing systems 
to emulate the possibly suboptimal excess barxlwidth 
sharing of GPS. 

It would thus be highly desirable to provide in a 
weighted fair queuing system emulating GPS. a method 
of achieving redistribution of unused bandwidth in a 
state-dependent manner, i.e.. ttiat reflects instantane- 
ous needs of the remaining bacMogged traffic flows. 

Summary of the Invention 

The instant invention is a modified approach to 
weighted fair queuing implementing an adaptive redistri- 
bution scheme. In such a scheme, each per connection 
flow is gi^ranteed its specified share of the link band- 
width with any excess bandwidth being adaptively redis- 
tributed. The scheme enables preservation of fair 
queuing's ability to provide worst case erxJ to end delay 
bounds and the schemes work like fair queuing when 
there is no excess bandwidth. The excess bandwidth 
may be distributed according to a different criteria. 
Examples of state dependent criteria are: 1) Longest 
delay first (LDF) that serves the f kjw with current long- 
est delay; 2) Le^ time to overflow (LTD) that serves the 
flow with minimum difference between maximum 
allowed delay and current delay: 3) Least time to over- 
flow with leaky buckets (LTD-LB) that sen/es the flow 
which would cause buffer overflow first if worst case 
arrivals happen. 

Advantageously, the adaptive bandwidth redistritxj- 
tion scheme always provides the minimum guarantees 
required for each of the connections and satisfies worst 



case fairness measures. 

The LDF policy uses excess bandwidth to reduce 
the variance of the delay distrbution wftich has the 
added benefit of reducing the playout buffer size for 
5 voice and video sources. Simulatiorts with video traces 
and with voice traffic shews that indeed this policy per- 
forms better than GPS without any sacrifrce of worst 
case guarantees. Since the delation from the maxi- 
mum allowed delay is not taken into account, flows with 
10 small delay bounds (like voice) get alnK>st no excess 
bandwidth in the presence of flows with large delay 
bounds. Impreciseness in assigning weights can result 
in these flows experiencing losses much more than 
f k)ws with large delay bounds. 
75 The LTO policy tries to minimize packet losses by 
assigning excess bandwidth under the assumption that 
the flow which is likely to overflow the quickest has the 
most instantaneous t)andwidth need. In doing so, it 
takes into account the current deviation of each flow 
20 from its maximum allowed delay Simulations with a mix 
of CBR, voice, and video sources with very different 
delay bounds show that this policy reduces losses for all 
classes as well as reduces the variance of delay for 
each class. The various features of novelty which char- 
ts acterize the invention are pointed out 

with particularity in the claims annexed to and form- 
ing a part of the disclosure. For a better understanding 
of the the invention, its operating advantages, and spe- 
cific objects attained by its use. reference should be had 
30 to the drawings and desaiptive matter in which there 
are illustrated and described preferred embodiments of 
the invention. 

Brief Description of Drawings 

35 

FIG. 1 is a general data flow diagram illustrating the 
features of the weighted fair queuing and a rate propor- 
tional Server. 

FIG. 2 illustrates the weighted fair queuing schedul- 
40 ing system of the invention. 

Detailed Description of the Invention 

Figure 2 is a general block diagram illustrating the 
45 fair queueing method with adaptive excess t>andwidth 
redistribution. The system consists of a set of queues 
100. a shaper device 30a...30i associated with each 
queue, and Weighted Fair Queueing Server or Rate 
Proportional Server 40 and a State Dependent Server 
so (SDS) 200. Packets are temporarily stored in the 
queues 20a....20i. A weight is assigned with each of 
these queues representing the portion of the output 
bandwidth that must be allocated to the packets arriving 
in each of the queues. The shaper device is fonwarding 
55 packets from the queues to the Weighted Fair Queueing 
Server with a rate exactly equal to the allocated. This 
scenario enables excess bandwidth to become availa- 
ble for redistribution. In accordance with the principles 
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of the invention, traffic from another source, i.e.. another 
queue, may be directed to the adaptive bandwidth redis- 
tribution mechanism, which is hereafter referred to as 
slate dependent scheduler 200. ft should be understood 
to skilled artisans that the hardware architectures for 5 
Implementation of the per-connection queues, shaper 
devices, and schedulers may vary, without cfiverting or 
straying from the principles of the Invention described 
herein. 

As described above, the shaper releases packets to 
the scheduler with a rate exactly equal to the allocated 
rate. TTie service offered by the shapers 30a... ,301 to the 
respective queue connections T during an interval of 
time is denoted by S^(x,t) in Figure 2. H is assumed that 
packets are transferred from the shaper to the sched- 
uler with infinite capacity. The service offered by the rate 
proportional scheduler 40 is denoted as Rj (x.t). 

Packets that have not become eligible for service 
remain In the corresponding connection queue in the 
shaper, while all the eligible packets wait lor service in 
the RPS Scheduler queue 40. Service is always pro- 
vided from the RPS scheduler queues 40, as long as 
packets are available there. When all of the scheduler 
queues are enpty, the state SDS 200 is invoked to 
select a packet from the queues 20a,..,20i for transmis- 
sion with the service offered to a connection not affect- 
ing the state of the shaper 30. As shown in Fig. 2, the 
service offered by the shaper 30 to the queue connec- 
tion T during an interval of time Is denoted by Dj(T.t) 
and the service offered by the state dependent sched- 
uler 200 is denoted as (x.t). 

The shaper device 30 can be implemented using a 
calendar queue that Is a well known mechanism to any 
skilled artisans. A description of the calendar queue 
mechanism for shaping has been presented In D. Stili- 
adis and A. Verma: A General Methodology for Design- 
ing Scheduling and Shaping Algorithms, in Proceedings 
of IEEE INFOCOM '97. 

The Rate Proportional Server (Weighted Pair 
Queueing Server) 40 can be implemented using any 
known fair queueing mechanisms of which there are 
several known to those skilled in the art. Examples are 
D. Stiliadis and A. Varma: Traffic Scheduling System 
and Method for Packet-Switched Networks, U.S. Patent 
Application, Serial No. 08/634.904, filed April 15.1996. 

At some point of time, if it Is determined tfiat the are 
no packets in the RPS scheduler, then this is an indica- 
tion of free bandwidth. Thus, packets may be served by 
the SDS 200 based on a state dependent manner. By 
state dependent manner, it is meant that the decision is 
based on the current state of some variat)les related to 
the system. Two exanrples of state dependent decisions 
are Longest Delay First and Least Time to Overflew. 

In the Longest Delay First, the scheduler SDS 200 
selects anx>ng the queues 20a,...20i. the one that has 
the packet that will see the longest delay if this queue is 
sensed with a rate equal to guaranteed. To do that, the 
scheduler assigns to each queue a delay value that is 
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equal to the size of the queue divided by the guaranteed 
rate. This value describes the delay that the last packet 
of this queue will see. The SDS 200 selects the queue 
with the maximum such value using any mecfianism for 
selecting the nnaximum value among a set of numbers. 
The selection mechanism is a well known method to 
any skilled artisan. 

In the Least Time to Overfk)w mechanism, the SDS 
scheduler 200 selects anx>ng the queues 20a.....20i 
that queue wtiich is likely to overflow in the shortest 
time. The mechanism assumes that a maximum size is 
associated with a queue. Every time a packet arrives or 
Is serviced by a queue, a packet-to-overflow variable 
counts the number of packets that can be added to the 
queue without exceeding its maximum capacity. A sec- 
ond variable, denoted as time-to-overflow. Indicates the 
expected time to overflow and is calculated by dividing 
the packet-to-overflow variable by the associated rate. 
The SDS 200 selects for transmission the queue with 
the minimum time-to-overflow variable. The minimum is 
determined by any of the well known mechanisms to 
anyone skilled in the art. Although the above variables 
explain two methods for calculating state 

dependent variables, other similar methods may be 
used. Based on the state dependent variables, the 
scheduler mechanism 200 will direct packets from the 
selected queues ever separate link 75 where the packet 
will be multiplexed by multiplexer element 350 arxJ for- 
warded to its next destination. 

It should be understood tfiat the property of worst 
case fairness is also satisfied In the system of the Inven- 
tion. In worst case fairness, the servicing of queues of 
different lengtfis at different allocated rates Is Inter- 
leaved such that a packet from a first queue is serviced 
at time t^ and the packet of a second queue is serviced 
at a time X2. In the Invention, the measure of worst ease 
fairness is satisfied, such that, the time interval \2'X^, In 
the worst case, is less than or bounded by a value that 
is not a function of the number of connections, txjt Is a 
function of the packet size of the longest queue. e.g.. 
the queue serviced at time t^ and the allocated rate of 
that longest queue. 

Thus, the property of worst case fairness is satis- 
fied by the implementation of the SDS 200 in the 
weighted fair queuing system of the invention. 

The foregoing merely illustrates the prindples of the 
present invention. Those skilled in the art will be atiie to 
devise various modifications, which although not explic- 
itly described or shown herein, embody the prindples of 
the invention and are thus within its spirit and scope. 

Claims 

1. Apparatus for routing packets In a communication 
network conrprlslng: 

a plurality of per-connectlon queues, each 
queue established for receiving packets from a 
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respective source and temporarily storing 
received packets before routing to a particular 
destination; 

weighted fair queuing scheduling means serv- 
icing packets from each of said plurality of per- s 
connection queues at guaranteed pre-allo- 
cated rates; 

means for sensing a presence or at>sence of 
packets in queues, said absence of packets in 
queues indicating availability of excess band- 10 
width; and, 

state dependent scheduling means for redis- 
tributing excess bandwidth upon sensing of 
queues absent said packets, said state 
dependent scheduling means servicing those is 
queues in accordance with a state variat)le cor- 
responding to a performance property of said 
queues. 

wherein delay and isolation properties 
for routing packets of respective queues is pre- 20 
served. 

2. Apparatus for routing packets in a communication 
network as claimed in Claim 1 , wherein a said per- 
formance property of a queue includes the queue 2s 
having the greatest amount of packets, said corre- 
sponding state variable being longest queue first. 

3. Apparatus for routing packets In a communication 
network as claimed in Claim 1 , wherein a said per- 30 
fbrmance property of a queue includes that queue 
having a packet waiting to be serviced in the long- 
est amount of time, said corresponding state varia- 
ble being longest delay first. 

35 

4. Apparatus for routing packets in a comnujnication 
network as claimed in Claim 1 , wherein a said per- 
formance property of a queue Includes the queue 
most likely to overflow buffer memory containing 
said queue, said corresponding state variable being 40 
most likely to overflow buffer. 

5. Apparatus for routing packets in a comnruinication 
network as claimed in Claim 1, wherein a worst 
case fairness measure for said connections is satis- <5 
fied. 

6. Method for routing packets in a communication net- 
work comprising: 

50 

establishing a plurality of per-connection 
queues, each queue for receiving packets from 
a respective source and temporarily storing 
received packets before routing to a particular 
destination; ss 
servicing packets from each of said plurality of 
per-connection queues at guaranteed pre-allo- 
cated rates; 



sensing a presence or absence of packets in 
queues, said absence of packets in queues 
incf eating availability of excess bandwidth; and, 
for redistr^xiting excess bandwidth upon sens- 
ing of queues ak>sent said packets, said state 
dependent scheduling means servicing those 
queues in accordance with a state varlatsle cor- 
responding to a performance property of sakJ 
queues, while preserving delay arxi Isolation 
properties for routing of said packets. 
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